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(54) Title: IMAGE RESTORATION OF A SINGLE-CHIP IMAGE SENSOR 
(57) Abstract 

The invention relates to a signal processing method for estimating 
luminance, for use with matrix patterned image sensors where elected sensor 
pixels (G), preferably green pixels, from which luminance-representative 
signals are taken, only occupy alternate horizontal and vertical positions in 
the matrix pattern. The method is designed for detecting luminance contour 
edges and/or stripes in an image so as to enable images obtained from the 
sensor to be enhanced. According to the method, each pixel site (He) which 
is at the centre of both a row and a column of five pixels and which is not 
the site of an elected (green) pixel is identified and for each such identified 
site (He) the signals output from the immediately adjacent four pixels (Gn, 
Ge, Gs, Gw) in said row and column are processed to establish both signal 
amplitude and pixel position. A luminance value for the pixel at the identified 
site (He) is evaluated from the amplitude values (G2, G3) of the median pair of 
said signal amplitudes in accordance with an algorithm where decision taking 
is based upon the location of the pixels from which the two signals (Gl, 
G2) of highest amplitude emanate. The algorithm is such that when the two 
signals (Gl, G2) of highest amplitude emanate from pixels in the same row or 
column a further signal processing step is undertaken to identify, in said row 
and column, the signal outputs from the two respective next adjacent pixels 
<ni, H4) to the pixels having the highest and lowest signal amplitudes (Gl, 
G4), and the signal output from the pixel at the identified site (He), to enable 
an amplitude comparison to be made according to the result of which one of 
the said median pair (G2, G3) is selected. Signal processing methods for use 
with colour, monochrome and mixed colour/monochrome image sensors are 
claimed, as well as a single-chip camera, and also a video camera system, in 
which the claimed signal processing methods are implemented. 
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IMAGE RESTORATION OF A SINGLE-CHIP IMAGE SENSOR 

The present invention relates to image restoration 
techniques for use with image sensors and in particular to 
5 signal processing methods for luminance estimation for use 
with matrix array image sensors of the type commonly 
incorporated in video cameras. 

The development of low-cost video cameras is driven by the 
10 existing Closed Circuit Television (CCTV) camera 

marketplace and by expected new markets, such as, for 
example, video conferencing. A major cost of these cameras 
is the image sensor chip whose cost is a monotonic function 
of sensor area. Two major goals of sensor designers are to 
15 improve sensitivity by increasing pixel area and improve 
resolution by increasing pixel count. Both of these steps 
have the disadvantage of increasing sensor area and hence 
sensor cost. If the sensor area is to remain constant 
then, until now, image resolution could only be increased 
20 at the cost of lowered sensitivity of the sensor and 

sensitivity could only be increased at the cost of lowered 
resolution. 

In one known type of video camera a colour image is formed 
25 on a single image sensor device which is spatially divided 
into many (typically 100,000 or more) pixels and each pixel 
is covered with a colour filter corresponding to one of a 
number of desired spectral components. These may typically 
be red, green and blue components or, for example, cyan, 
50 magenta and yellow. Further colour sets are also sometimes 
used. 

Many different types of colour matrix patterns are 
possible. The signal outputs of all the pixels are 
5 combined by means of a reading operation at the sensor 

device output. Many image restoration techniques have been 
developed for combining the signals from individual pixels 
in the matrix array into a composite image representation 
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which presents a u real"-looking image to the human eye. One 
known technique involves superimposing a highpass-f iltered 
or "edge-extracted" luminance signal onto lowpass-f iltered 
or "spatially smoothed" component colour signals (e.g. red, 
5 green, blue) obtained from the pixel matrix. Such image 
restoration techniques can enhance the perceived resolution 
or "sharpness" of the video camera image. 

One problem associated with image restoration is obtaining 
10 an accurate luminance signal from the sensor. It is known 
that the green spectral component is closely related to the 
luminance response of the human eye. It has therefore been 
known to use the signal outputs of green pixels, or 
alternatively another colour of pixel whose colour is 
representative of luminance, to obtain the necessary 
luminance signal. As the chosen colour of pixels (e.g. 
green) only account for some of the pixels in a colour 
image sensor array, however, it is desirable to estimate 
luminance at the remaining pixel sites. 

It is an object of the present invention to provide a 
signal processing method for accurately estimating 
luminance for use in matrix array colour image sensors. 

US 4 630 307 describes a signal processing method for 
estimating luminance for use in matrix array sensors having 
luminance-sensing pixels arranged in a chequerboard 
pattern. The method uses signals obtained from luminance- 
sensing pixels to estimate luminance values at pixels in 
the matrix array which are not capable of sensing 
luminance. 

In the following text the term "row" is defined as a 
horizontal row and the term "column" is defined as a 
vertical row. 
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According to a first aspect, the present invention 
comprises a signal processing method for estimating 
luminance, for use with matrix patterned image sensors 
where elected sensor pixels from which luminance- 
5 representative signals are taken only occupy alternate 
horizontal and vertical positions in the matrix pattern, 
the method comprising 

identifying each pixel site which is at the centre of both 
a row and a column of five pixels and which is not the site 
10 of a said elected sensor pixel, and for each such 

identified site processing the signals output from the 
immediately adjacent four pixels in said row and column to 
establish both signal amplitude and pixel position and 
evaluating a luminance value for the pixel of the 
15 identified site from the amplitude values of the median 
pair of said signal amplitudes in accordance with an 
algorithm where decision taking is based upon the location 
of the pixels from which the two signals of highest 
amplitude emanate, the algorithm being such that when the 
20 two signals of highest amplitude emanate from pixels in the 
same row or column a further signal processing step is 
undertaken to identify, in said row and column, the signal 
outputs from the two respective next adjacent pixels to the 
pixels having the highest and lowest signal amplitudes, and 
25 the signal output from the pixel at the identified site, to 
enable an amplitude comparison to be made according to the 
result of which one of the said median pair is selected. 

One advantage of the signal processing method according to 
30 the first aspect of the invention is that it utilizes 
signal outputs from pixels in said row and column which 
pixels are next adjacent to the highest and lowest 
amplitude ones of said four pixels immediately adjacent the 
identified site. 

35 

Preferably, the selected one of said median pair is the 
higher amplitude one where the signal output from the next 
adjacent pixel to said pixel having the highest signal 
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amplitude is closer in amplitude to the signal output from 
the pixel at the identified site than the signal output 
from the next adjacent pixel to said pixel having the 
lowest signal amplitude. Preferably, also, the selected 
one is the lower amplitude one of said median pair where 
the signal output from the next adjacent pixel to said 
pixel having the lowest signal amplitude is closer in 
amplitude to the signal output from the pixel at the 
identified site than the signal output from the next 
adjacent pixel to said pixel having the highest signal 
amplitude. 

According to a second aspect the present invention 
comprises a signal processing method for estimating 
luminance for use with matrix array colour image sensors 
which elected luminance-sensing pixels capable of producing 
signals representative of luminance are arranged in a 
chequerboard pattern with colour sensing pixels which are 
capable of producing signals representative of colour, the 
method comprising the steps of: ranking the signal values 
representative of luminance obtained from a cluster of four 
luminance-sensing pixels surrounding a first colour-sensing 
pixel, analyzing the spatial distribution of said ranked 
signal values so as to detect the presence of any diagonal 
luminance contour edge, horizontal luminance contour stripe 
or vertical luminance contour stripe at said first colour- 
sensing pixel, and determining an estimated discrete signal 
value representative of luminance for said first colour- 
sensing pixel based on one or more middle ranked signal 
values in a colour-enhancing fashion dependent upon, 
firstly, whether any diagonal luminance contour edge is 
detected and, secondly, whether any horizontal or vertical 
luminance contour stripe which is detected is interpreted 
as a dark stripe on a light background or a light stripe on 
a dark background, wherein when a horizontal or vertical 
luminance contour stripe is detected, the signal values 
representative of colour obtained from a plurality of said 
colour-sensing pixels in the vicinity of said cluster of 
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four luminance-sensing pixels are compared so as to 
determine whether said detected horizontal or vertical 
luminance contour stripe is interpreted as a dark stripe on 
a light background or a light stripe on a dark background. 

5 

One advantage of the signal processing method according to 
the second aspect of the invention is that it utilizes 
colour signal values from pixels other than said elected 
luminance-sensing pixels to influence the estimated 
10 luminance signal value for said first colour-sensing pixel. 

Preferably, said plurality of colour-sensing pixels in the 
vicinity of said cluster of four luminance-sensing pixels 
includes said first colour-sensing pixel, a second colour- 

15 sensing pixel disposed within the detected luminance 

contour stripe and a third colour-sensing pixel disposed 
outwith the detected luminance contour stripe. This 
increases the likelihood of accurately detecting and 
enhancing horizontal and vertical luminance contour stripes 

20 occurring in the image. The method thus enables real-time 
images obtained from the image sensor to be enhanced so as 
to improve the perceived resolution of the sensor to the 
human eye. 

25 The estimated discrete signal value representative of 
luminance is preferably based on the mean of the middle 
ranking signal values when a diagonal contour edge is 
detected at first colour-sensing pixel. 

30 Advantageously, said estimated discrete value is based on a 
corresponding one of the two middle ranking values when a 
stripe detected at said first colour-sensing pixel is 
interpreted as a dark stripe on a light background or as a 
light stripe on a dark background. Preferably said 

35 corresponding one of the two middle ranking signal values 
is the lower ranked one when a dark stripe on a light 
background is detected and is the higher ranked one when a 
light stripe on a dark background is detected. 
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A diagonal contour edge is preferably detected when the two 
highest ranking signal values are obtained from luminance- 
sensing pixels disposed diagonally adjacent one another. 

A horizontal or vertical contour stripe is preferably 
detected when the two highest ranking signal values are 
obtained from luminance-sensing pixels disposed 
horizontally or vertically opposite one another. 

Where a horizontal or vertical stripe is detected, the 
signal value obtained from said first colour-sensing pixel 
is preferably compared with the signal values obtained from 
second and third colour-sensing pixels disposed 
respectively vertically and horizontally opposite said 
first colour-sensing pixel and immediately adjacent to the 
luminance-sensing pixels having the highest and lowest 
ranked signal values. Conveniently, if the signal value of 
said first colour-sensing pixel is closest in value to the 
signal value of the pixel immediately adjacent said highest 
ranked signal value pixel the stripe is interpreted as a 
light stripe on a dark background and if the signal value 
of the first colour-sensing pixel is closest in value to 
the signal value pixel of the pixel immediately adjacent 
said lowest ranked signal value pixel the stripe is 
interpreted as a dark stripe on a light background. 

Said elected luminance-sensing pixels may be pixels which 
are capable of producing signals representative of both 
luminance and colour. 

The signal processing steps of the method according to the 
second aspect of the invention are preferably repeated for 
every colour-sensing pixel in the matrix array sensor which 
is not also an elected luminance-sensing pixel and which is 
at the centre of both a row and a column of five pixels. 
In this manner a respective estimated discrete signal value 
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representative of luminance is determined for each such 
colour-sensing pixel in the matrix array. 

The estimated luminance signal value obtained from each 
5 such colour-sensing pixel in the array may provide a 
luminance signal for a luminance channel of a camera 
incorporating the image sensor array. Said estimated 
luminance channel signal may be highpass- filtered to obtain 
an "edge-extracted" luminance signal. Alternatively, said 
10 estimated luminance signal may be lowpass-f iltered and 

further processed to produce a signal which approximates to 
an "edge-extracted" luminance signal. 

Signals representative of colour obtained from colour- 
15 sensing pixels" in the array may be processed to obtain 

lowpass-filtered colour channel signals in a colour camera 
which incorporates the sensor array. Where said elected 
luminance-sensing pixels also sense colour, the signals 
from the elected luminance-sensing pixels in the array may 
20 also be processed to obtain a lowpass-filtered signal for 
an additional colour channel. The "edge-extracted" 
luminance signal may be superimposed on said colour channel 
signals to obtain full colour and luminance information for 
every pixel which is used to produce a contour-enhanced 
25 (i.e. improved resolution) image to the human-eye. 

The elected luminance-sensing pixels are preferably green 
pixels. The colour-sensing pixels are preferably red and 
blue pixels. Alternatively, the matrix array is comprised 
30 of magenta, cyan and yellow pixels. Other colour sets are 
also possible. 

Where the matrix array is comprised of red, green and blue 
pixels, the pixel signals are read into respective red, 
35 green and blue colour channels. The three colour channel 
signals are preferably processed to obtain lowpass- 
filtered, "spatially-smoothed" respective red, green and 
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blue channel signals. In this way, red, green and blue 
colour information is obtained for every pixel in the 
array . 

5 Where green pixels are the elected luminance-sensing 

pixels, the green signal values are preferably used in raw, 
uncorrected colour form. 

The matrix array image sensor may, alternatively, comprise 
10 a mixed colour and monochrome array in which monochrome, 
luminance-sensing pixels are arranged in a cheguerboard 
pattern with colour-sensing pixels. 

According to a third aspect, the invention comprises a 

15 signal processing method for estimating luminance, for use 
with matrix patterned sensors where elected sensor pixels 
from which luminance-representative signals are taken only 
occupy alternate horizontal and vertical matrix element 
positions in the matrix pattern, 

20 the method comprising 

identifying each matrix element site which is at the centre 
of both a row and a column of seven matrix elements and 
which is not the site of a said elected sensor pixel, and 
for each such identified site processing the signals output 

25 from the immediately adjacent four pixels in said row and 
column to establish both signal amplitude and pixel 
position and evaluating a luminance value for the matrix 
element of the identified site from the amplitude values of 
the median pair of said signal amplitudes in accordance 

30 with an algorithm where decision taking is based upon the 
location of the pixels from which the two signals of 
highest amplitude emanate, the algorithm being such that 
when the two signals of highest amplitude emanate from 
pixels in the same row or column a further signal 

35 processing step is undertaken to identify outputs in said 
row and column from the two pixels respectively next 
adjacent to the pixels having the highest and lowest signal 
amplitudes, and to identify the mean of the signal outputs 



Copied from.^9-^-?-00-9n-06717./2005 



WO 97/35438 



-9- 



PCT/GB97/00599 



from the eight pixels diagonally adjacent to, and arranged 
around, said four pixels in said row and column, to enable 
an amplitude comparison to be made according to the result 
of which one of the said median pair is selected. 

5 

According to a fourth aspect the invention comprises a 
signal processing method for estimating luminance for use 
in a matrix array monochrome image sensor in which elected 
luminance-sensing pixels capable of producing signals 
10 representative of luminance are arranged in a chequerboard 
pattern with non- luminance-representative matrix array 
elements which do not produce signals, the method 
comprising the steps of: ranking the signal values 
representative of luminance obtained from a cluster of four 
15 luminance-sensing pixels surrounding a first non- luminance- 
representative matrix element, analyzing the spatial 
distribution of said ranked signal values so as to detect 
the presence of any diagonal luminance contour edge, 
horizontal luminance contour stripe or vertical luminance 
20 contour stripe at said first non- luminance-representative 
matrix element, and determining an estimated discrete 
signal value representative of luminance for said non- 
luminance-representative matrix element based on one or 
more middle ranked signal values in a contour-enhancing 
25 fashion dependent upon, firstly, whether any diagonal 

luminance contour edge is detected and, secondly, whether 
any horizontal or vertical luminance contour stripe which 
is detected is interpreted as a dark stripe on a light 
background or a light stripe on a dark background, wherein, 
30 when a horizontal or vertical luminance contour stripe is 
detected the signal values representative of luminance 
obtained from at least second and third luminance-sensing 
pixels disposed respectively within and outwith the 
detected luminance contour stripe are compared with the 
35 mean of the signal values obtained from a plurality of 
further luminance-sensing pixels in the vicinity of said 
cluster of four luminance-sensing pixels so as to determine 
whether said horizontal or vertical stripe is interpreted 
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as a dark stripe on a light background or a light stripe on 
a dark background. 

An advantage of this fourth method is that the signal value 
5 from at least one pixel disposed within the detected 
luminance contour stripe is compared with signal values 
from pixels disposed outwith said stripe so as to determine 
whether the stripe should be interpreted as a dark stripe 
on a light background or a light stripe on a dark 
10 background . 

Preferably said plurality of further luminance-sensing 
pixels comprise eight pixels, each pixel being disposed 
diagonally adjacent one of said cluster of four luminance- 
15 sensing pixels surrounding said first non-luminance 
representative matrix element. 

Advantageously said second and third luminance-sensing 
pixels are disposed horizontally and vertically opposite 
20 said first non-luminance representative matrix element and 
adjacent to the two of the cluster of four luminance- 
sensing pixels which have the highest and lowest ranked 
signal values. 

25 The steps of the signal processing method according to the 
fourth aspect of the invention may be repeated for each 
non- luminance representative matrix element in the matrix 
array which is at the centre of both a row and a column 
containing seven matrix elements. 

30 

According to a fifth aspect, the invention provides a 
camera comprising a matrix array image sensor and signal 
processing means for carrying out the signal processing 
method according to any of the above-mentioned first, 
35 second, third and fourth aspects of the invention. The 
image sensor may be a Charge Coupled Device (CCD) or CMOS 
array . 
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Preferred embodiments of the invention will now be 
described by way of example only and with reference to the 
accompanying drawings in which: 

5 Fig. 1 is a schematic illustration of a 5 x 5 portion of a 
colour (RGB) matrix image sensor which may be used with a 
signal processing method for luminance estimation according 
to the invention; 

Fig. 2 is a schematic illustration of a 5 x 5 pixel 
10 template centered on a non-Green pixel, He, of the image 
sensor of Fig. 1; 

Figs. 3 (a) and (b) illustrate two possible situations 

where a horizontal or vertical stripe is detected at the 

non-Green pixel, He, of Fig. 2; 
15 Fig. 4 is a flow diagram illustrating schematically a 

luminance estimation method according to the invention; 

Fig. 5 is a schematic illustration of a 4 x 4 portion of a 

mixed colour and monochrome matrix image sensor; 

Fig. 6 is a schematic illustration of a portion of a 
20 monochrome image sensor array which may be used with a 

further method according to the invention; 

Fig. 7 is a block diagram illustrating schematically a 

single-chip colour video camera in which signal processing 

methods according to the invention are carried out; and 
25 Fig. 8 is a block diagram illustrating schematically a 

video camera system 40 incorporating the chip camera of 

Fig. 7. 

Fig. l illustrates schematically a 5 x 5 pixel portion 1 of 
30 a single image sensor device having a Red (R) , Green (G) , 
Blue (B) colour matrix pattern as disclosed in US 3 971 065 
(Bayer) . The pattern contains more Green (G) pixels than 
Red (R) or Blue (B) because Green is the principal, primary 
colour component of luminance. The Green pixels are 
35 disposed in a chequerboard pattern with the non-green (i.e. 
R,B) pixels, the green pixels (G) thus occupying alternate 
horizontal and vertical positions in the matrix. 
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The sensor device is spatially divided into typically 
100,000 or more pixels and each pixel is covered with a 
colour filter corresponding to the desired spectral 
components. The device is incorporated in an electric 
camera (see Figs. 7 and 8) in which signal outputs from the 
pixels are read out at an array output of the sensor device 
to provide three component colour channels, as in a normal 
electronic colour camera. 

The component colour channels are interpolated to provide 
missing data by lowpass-f iltering using a square area 
template, typically 5x5 pixels. Although Green <Jata is 
more plentiful than Red or Blue, and accordingly more Green 
pixels are covered by the lowpass-f ilter template than Red 
or Blue, the Green colour-signal is essentially treated 
identically to Red and Blue. In each case, the component 
colour channel represents a lowpass-f iltered (spatially 
smoothed) version of the ideal component colour channel 
response. 

In addition to the three component colour channels normally 
present in an electronic colour camera, we form a fourth 
channel representing an estimate of luminance. This is 
achieved using the following process. 

The Green pixels (G) are elected to be "luminance-sensing" 
pixels and Green pixel signals are taken to be 
representative of luminance as well as being representative 
of Green colour. Green is the dominant contributor to 
luminance, as perceived by the human eye, and is the most 
densely sampled colour component in the matrix pattern of 
the Fig. 1 sensor device. It is used in a unique fashion 
when estimating luminance. The luminance estimate is 
constructed by interpolating non-Green pixels in a 
luminance contour-enhancing fashion, complementing the 
contour-suppressing interpolation of the colour channel 
processing. 
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The operations described below are symmetrical in an 
orthogonal sense, that it to say transformations such as 
mirroring or rotation through multiples of 90 degrees do 
not change their result. With no loss of generality, we 
5 associate compass direction N (north) with the top of the 
image . 

Sites where Green is present require no processing. In 
this case luminance is given by the centre-pixel output, 
10 Gc. A luminance estimate for each "missing Green" (i.e. Red 
(R) or Blue (B) site) which is at the centre of both a row 
and a column of five pixels is obtained using a 5 x 5 
template 2 centred on the "missing Green" site, He, as 
illustrated schematically in Fig. 2, wherein neighbours to 

15 the N,S,E and W are known to be Green. These appear as Gn, 
Ge, Gs, and Gw in Fig. 2. We identify non-Green pixels in 
the 5 x 5 pixel vertical cross pattern of Fig. 2 as Hn, He, 
Hs, and Hw, being the far-neighbours of Gn, Ge, Gs and Gw 
respectively. These H pixels will all be either Red or 

20 Blue depending on the colour of the centre pixel He. 

In the first method described here, we first sort the four 
G neighbours by descending order of value (i.e. amplitude) 
into Gl (max), G2, G3 , G4 (min) . If Gl and G2 form a 
25 diagonally-adjacent pair then we infer that the image 

contains a local diagonal edge (of a luminance contour) and 
the missing luminance value is estimated or "restored" as 
the average of the median pair, G2 and G3. in this case 
the values of H pixels are unused. 

30 

If G2 and G2 are horizontally or vertically opposed across 
the "missing Green" pixel, He, for example as shown in Figs. 
3 (a) and (b) , we infer that the image contains part of a 
horizontal or vertical luminance contour line or "stripe". 
55 To determine whether the stripe is a dark stripe on a light 
background or a light stripe on a dark background we 
examine the far neighbours Hi and H4 of Gl and G4 
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respectively to determine which of these is closer in value 
to He. If HI is closer then we infer that we have a light 
stripe on a dark background and we restore luminance at the 
missing Green site, He, at a value equal to G2, i.e. the 

5 higher of the two middle ranked values G2, G3. If H4 is 
closer we infer that we have a dark stripe on a light 
background and we restore luminance at He at a value equal 
to G3 i.e. the lower of the two middle ranked values. In 
each case these median values (G2 , G3) are generally safer 

10 alternatives to the maximum or minimum values (Gl, G4) . 

Fig. 4 illustrates a decision flow chart which summarizes 
the luminance estimation method applied so as to step 
through the image, examining every pixel site which is at 

15 the centre of both a row and a column of five pixels. The 
first branch 3 depends on whether the centre-pixel site 
contains a Green pixel or not. If yes (Y) , the green 
signal Gc is output. If no (N) , Gn, Gc, Gs, and Gw are 
sorted into Gl, G2, G3, G4 (see box 4) . The second branch 

20 5 depends on whether a diagonal split is detected, i.e. Gl 
is diagonally adjacent to G2. The third branch 7 is 
dependent on whether a "dark choice" is made, i.e. far- 
neighbouring H4 is closer than HI in value to He. 

25 Gl and G2 being horizontally or vertically opposed across 
the "missing Green" pixel, He, may also indicate 
respectively the presence of a sharp horizontal or vertical 
luminance contour edge (rather than a stripe) in the image. 
The second branch of the decision process, i.e. the "dark 

30 choice" of Fig. 4, produces equally acceptable restored 
luminance values for the missing-Green pixel in this 
scenario. Flat, i.e. relatively constant luminance, areas 
of the image will also be detected as either diagonal 
edges, or vertical or horizontal stripes, using the above- 

35 described processing method, which produces acceptable 
restored luminance values for such flat areas. 
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The estimated luminance signal obtained for a non-Green 
pixel sited at the centre of both a row and a column of 
five pixels provides an estimated luminance signal for the 
fourth i.e. luminance channel of the camera. This signal 
5 then has subtracted from it a spatially smoothed version of 
the luminance-representative signal, this being the 
particular output of a 5 x 5 square area lowpass-f iltering 
template corresponding to the colour used for luminance 
sensing (in this case Green). This technique, sometimes 

10 known as "un-sharp masking", obtains a signal which 

approximates to a highpass-f iltered (i.e. "edge-extracted") 
luminance signal. The "edge-extracted" signal is then 
superimposed on to the spatially-smoothed colour channel 
signals obtained over the same 5x5 pixel area so as to 

15 produce a Red, Green, Blue (RGB) pixel triplet including 
full colour and luminance information. RGB pixel triplet 
signals obtained in this way enable the camera to produce a 
contour-enhanced image which is more resolved, and thus 
more pleasing, to the human eye, than the image obtained 

20 purely from the spatially smoothed colour channel signals. 

The above-described luminance estimation signal processing 
method restores luminance values to most pixel sites in the 
sensor array and gives subjectively excellent "end-image" 
25 results for most real scenes. 

The method is particularly successful in detecting and 
restoring luminance contours of diagonal edge, vertical 
stripe and horizontal stripe forms, these having been 
30 particularly difficult to detect and restore accurately 
using previously known luminance estimation/ interpolation 
techniques. 

Conveniently, in its "raw" form Green contains components of 
35 Red and Blue present due to imperfections of the Green 

filter spectral response, which can improve its utility as 
an estimate of luminance. Although it is normal practice 
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to compensate electronically for incorrect colour 
rendering, it is better to apply luminance estimation 
processing before colour-compensation processing on Green. 

5 it should be noted that colour combinations other than RGB 
may be used. The above procedure may produce similarly 
good results for these alternative combinations provided 
the colour that is elected for the luminance-sensing pixels 
in place of Green is representative of luminance to the 

10 human eye. For example, a yellow, cyan, magenta colour 
combination could be used. 



The image sensor device is typically a CCD or CMOS array 
sensor. The production process for both types of sensor is 

15 well disposed to deposition of the colour filters when the 
products are in silicon-wafer form. This enhances cost 
benefit. Signal processing circuitry for carrying out the 
above-described luminance estimation according to the flow 
chart of Fig. 4 can be provided in the image sensor chip, 

20 if desired. Alternatively, or additionally, separate 
electronic circuitry can be provided in the camera. 

Fig. 5 shows schematically a portion 8 of a mixed 
monochrome and colour image sensor in which monochrome 

25 (e.g. white, unfiltered) pixels (Y) are arranged in a 
chequerboard pattern with coloured pixels which may be 
coloured in accordance with any desired colour set but in 
Fig. 5 are shown as Red(R), Green (G) , Blue(B) pixels. The 
luminance estimation method described in relation to Figs. 

30 l to 4 can also be applied to this type of sensor array by 
electing the monochrome pixels as the "luminance-sensing" 
pixels. Colour restoration is not applied in this case. 

The above-described luminance estimation method can also be 
35 adapted for application to a monochrome matrix array image 
sensor. Fig. 6 shows schematically a portion 9 of a 
monochrome sensor array in which photosensitive pixels (P) , 
shown in white, are arranged in a chequerboard pattern with 
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non-photosensitive areas (N) , shown shaded, which are 
occupied by overhead (light-insensitive) circuitry. 

A cluster of four pixels Yn, Ye, Ys, Yw surrounding a non- 
5 photosensitive centre site C take the place of Gn, Ge, Gs, 
Gw respectively in Fig. 2. These are ranked in descending 
value order as Yl (max), Y2, Y3, Y4 (min) and are used to 
obtain a median average (the mean of Y2 and Y3) for 
estimated luminance at centre site C in cases where a 

10 diagonal split is detected. The principal difference 

between the monochrome technique for Fig. 6 and the colour 
technique for Fig. 2 is in the "dark choice" step, required 
in cases where a diagonal split is not detected. Here, in 
order to infer whether the image contains a dark stripe on 

15 a light background or a light stripe on a dark background, 
the pixels Xn, Xe, Xs, Xw, which are the far neighbours of 
Yn, Ye, Ys, Yw in a vertical cross pattern centered on C, 
take the place of Hn, He, Hs, Hw in Fig. 2 respectively. 

20 He in Fig. 2 is, however, unobtainable in this case as 
centre-site C does not contain a pixel. Instead, we 
calculate the average (i.e. mean) of the eight ring- 
surrounding pixels, Xc, which are located diagonally 
adjacent to Yn, Ye, Ys, Yw as shown in Fig. 6. The far 

25 neighbours XI and X4 of Yl (max) and Y4 (min) respectively 
are compared with the mean, M, of the eight ring- 
surrounding pixels, Xc. If XI is closer in value than X4 
to M we infer a dark stripe on a light background and 
luminance is estimated or "restored" at centre-site C as Y3 . 

30 if X4 is closer in value than XI to M we infer a light 
stripe on a dark background and luminance is estimated or 
restored at C as Y2. 

By using this "monochrome" luminance estimation method, the 
pixels representing luminance can be maximized in size (50% 
35 of the number of pixels present in the colour array of Fig. 
1 having effectively been discarded in the monochrome 
version of the array) to enable the realization of a 
monochrome video camera of increased sensitivity without 
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the resulting unacceptable loss of image resolution, due to 
the lower pixel count, which would be experienced if the 
luminance estimation technique were not employed. 

5 The skilled reader will be aware of various suitable ways 
of implementing the necessary electronic circuitry to 
perform the signal processing operations illustrated by 
Fig. 4. This may be achieved, for example, in integrated 
circuitry which may, as afore-mentioned, be provided on the 
10 image sensor chip. Fig. 7 is a schematic architectural 
overview of a single-chip colour video camera 20. The 
camera 20 incorporates a sensor subsystem 21 comprising a 
control unit 22, and image sensor array 23, and a readout 
unit 24. Sensor output S, is digitised (block A/D) in a 
15 processor subsystemn 25 and stored in a 5-line memory 26. 
As a new pixel is written into memory, a 5 x 5 pixel block 
is read out from 5-pixel registers 27 and processed in two 
channels, the upper channel including a lowpass filter 28 
and processor block 29 acting on colour date. The lower 
20 channel includes signal processing apparatus at block 30 

for carrying out the luminance estimation signal processing 
method already described. A subtractor circuit 35 is used 
to perform the un-sharp masking process in which the 
spatially-smoothed version of the Green signal is 
25 subtracted from the luminance estimate signal. After 
further processing at block 31 to obtain the "edge- 
extracted" luminance signal, the colour and luminance 
signals are recombined using adding circuit 32 to produce 
the final RGB pixel triplet. A timing generator 3 3 is also 
30 provided for the processor subsystem 25 as shown in Fig. 7. 

Fig. 8 shows an NTSC or PAL (phase alternate line) video 
camera system 40 incorporating the chip camera 20 of Fig. 
7. The system 40 includes an NTSC or PAL encoder 42 which 
35 receives the RGB video signal V and a synchronising signal 
Z from the camera 20. The system also includes a resonant 
crystal 44 which functions as a master clock for driving 
the camera 20. 
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CLAIMS 

1. A signal processing method for estimating luminance, for 
use with matrix patterned image sensors where elected 
5 sensor pixels (G) from which luminance-representative 
signals are taken only occupy alternate horizontal and 
vertical positions in the matrix pattern, 
the method comprising 

identifying each pixel (He) which is at the centre of both 

10 a row and a column of five pixels and which is not the site 
of a said elected sensor pixel (G) , and for each such 
identified site (He) processing the signals output from the 
immediately adjacent four pixels (Gn, Ge, Gs, Gw) in said 
row and column to establish both signal amplitude and pixel 

15 position and evaluating a luminance value for the pixel of 
the identified site from the amplitude values (G2, G3) of 
the median pair of said signal amplitudes in accordance 
with an algorithm where decision taking is based upon the 
location of the pixels from which the two signals (Gl, G2) 

20 of highest amplitude emanate, the algorithm being such that 
when the two signals (Gl, G2) of highest amplitude emanate 
from pixels in the same row or column a further signal 
processing step (7) is undertaken to identify, in said row 
and column, the signal outputs from the two respective next 

25 adjacent pixels (HI, H4) to the pixels having the highest 
and lowest signal amplitudes (Gl, G4) , and the signal 
output from the pixel at the identified site (He) , to 
enable an amplitude comparison to be made according to the 
result of which one of the said median pair (G2, G3) is 
30 selected. 

2. A signal processing method according to claim 1, wherein 
the selected one of said median pair (G2, G3) is the higher 
amplitude one (G2) where the signal output from the next 
35 adjacent pixel (HI) to said pixel having the highest signal 
amplitude (Gl) is closer in amplitude to the signal output 
from the pixel at the identified site (He) than the signal 
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output from the next adjacent pixel (H4) to said pixel 
having the lowest signal amplitude (G4) . 

3. A signal processing method according to claim 1 or claim 
5 2, wherein the selected one of said median pair (G2, G3) is 

the lower amplitude one (G3) where the signal output from 
the next adjacent pixel (H4) to said pixel having the 
lowest signal amplitude (G4) is closer in amplitude to the 
signal output from the pixel at the identified site (He) 
10 than the signal output from the next adjacent pixel (HI) to 
said pixel having the highest amplitude (Gl) . 

4. A signal processing method for estimating luminance for 
use with matrix array colour image sensors in which elected 

15 luminance-sensing pixels (G) capable of producing signals 
representative of luminance are arranged in a chequerboard 
pattern with colour sensing pixels (R, B) which are capable 
of producing signals representative of colour, the method 
comprising the steps of: ranking the signal values (Gl, G2, 

20 G3, G4) representative of luminance obtained from a cluster 
of four luminance-sensing pixels (Gn, Ge, Gs, Gw) 
surrounding a first colour-sensing pixel (He) , analysing 
the spatial distribution of said ranked signal values (Gl, 
G2, G3, G4) so as to detect the presence of any diagonal 

25 luminance contour edge, horizontal luminance contour stripe 
or vertical luminance contour stripe at said first colour- 
sensing pixel (He) , and determining an estimated discrete 
signal value representative of luminance for said first 
colour-sensing pixel based on one or more middle ranked 

30 signal values (G2, G3) in a contour-enhancing fashion 
dependent upon, firstly, whether any diagonal luminance 
contour edge is detected and, secondly, whether any 
horizontal or vertical luminance contour stripe which is 
detected is interpreted as a dark stripe on a light 

35 background or a light stripe on a dark background, wherein 
when a horizontal or vertical luminance contour stripe is 
detected the signal values representative of colour 
obtained from a plurality of said colour-sensing pixels (R, 
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B) in the vicinity of said cluster of four luminance- 
sensing pixels (Gn, Ge, Gs, Gw) are compared so as to 
determine whether said detected horizontal or vertical 
luminance contour stripe is interpreted as a dark stripe on 
5 a light background or a light stripe on a dark background. 

5. A signal processing method according to claim 4, wherein 
said plurality of colour-sensing pixels in the vicinity of 
said cluster of four luminance-sensing pixels (Gn, Ge, Gs, 

10 Gw) includes said first colour-sensing pixel (He) , a second 
colour-sensing pixel (H1,H4) disposed within the detected 
luminance contour stripe and a third colour-sensing pixel 
(H4,H1) disposed outwith the detected luminance contour 
stripe. 

15 

6. A signal processing method according to claim 4 or claim 
5, wherein the estimated discrete signal value 
representative of luminance is based on the mean of the 
middle ranking signal values (G2, G3) when a diagonal 

20 contour edge is detected at said first colour-sensing pixel 
(He) . 

7. A signal processing method according to any of claims 4 
to 6, wherein said estimated discrete signal value is based 

25 on a corresponding one of the two middle ranking values 
(G2, G3) when a stripe detected at said first colour- 
sensing pixel (He) is interpreted as a dark stripe on a 
light background or as a light stripe on a dark background. 

30 8. A signal processing method according to claim 7, wherein 
said corresponding one of the two middle ranking signal 
values (G2, G3) is the lower ranked one (G3) when a dark 
stripe on a light background is detected and is the higher 
ranked one (G2) when a light stripe on a dark background is 

35 detected. 

9. A signal processing method according to any of claims 4 
to 8, wherein a diagonal contour edge is detected when the 
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two highest ranking signal values (Gl, G2) are obtained 
from luminance-sensing pixels (G) disposed diagonally 
adjacent one another. 

10. A signal processing method according to any of claims 4 
to 9, wherein a horizontal or vertical contour stripe is 
detected when the two highest ranking signal values (Gl, 
G2) are obtained from luminance-sensing pixels (G) disposed 
horizontally or vertically opposite one another. 

11. A signal processing method according to any of claims 4 
to 10, wherein, where a horizontal or vertical stripe is 
detected, the signal value obtained from said first colour 
sensing pixel (He) is preferably compared with the signal 
values obtained from second and third colour-sensing pixels 
(Hi, H4) disposed respectively vertically and horizontally 
opposite said first colour-sensing pixel (He) and 
immediately adjacent to the luminance-sensing pixels having 
the highest and lowest ranked signal values (Gl, G4) , and 
if the signal value of said first colour-sensing pixel (He) 
is closest in value to the signal value of the pixel (HI) 
immediately adjacent to said highest-ranked signal value 
(Gl) the stripe is interpreted as a light stripe on a dark 
background, and if the signal value of said first colour- 
sensing (He) is closest in value to the signal value of the 
pixel (H4) immediately adjacent to said lowest-ranked 
signal value (G4) the stripe is interpreted as a dark 
stripe on a light background. 

12. A signal processing method according to any of claims 4 
to 11, wherein the steps of the method are repeated for 
every colour-sensing pixel (R,B) in the matrix array sensor 
which is not also an elected luminance-sensing pixel (G) 
and which is at the centre of both a row and a column of 
five pixels. 

13. A signal processing method according to any of claims 4 
to 12, wherein the estimated discrete signal value 
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representative of. luminance obtained from each said colour 
sensing pixel (R, B) in the array is used to provide a 
luminance signal for a luminance channel (35, 31) of a 
camera (20) incorporating the image sensor array (23), 
5 which luminance signal is further processed to obtain an 
"edge-extracted" luminance signal which is, in turn, 
superimposed on lowpass filtered colour channel signals (R, 
G, B) in the camera (20) so as to provide luminance 
information for every pixel in the array. 

10 

14. A signal processing method for estimating luminance, 
for use with matrix patterned sensors where elected sensor 
pixels (P) from which luminance-representative signals are 
taken only occupy alternate horizontal and vertical matrix 
15 element positions in the matrix pattern, 
the method comprising 

identifying each matrix element site (C) which is at the 
centre of both a row and a column of seven matrix elements 
and which is not the site of a said elected sensor pixel 
20 (P) , and for each such identified site (C) processing the 
signals output from the immediately adjacent four pixels 
(Yn, Ye, Ys, Yw) in said row and column to establish both 
signal amplitude and pixel position and evaluating a 
luminance value for the matrix element of the identified 
25 site (C) from the amplitude values of the median pair (Y2, 
Y3) of said signal amplitudes in accordance with an 
algorithm where decision taking is based upon the location 
of the pixels from which the two signals (Yl, Y2) of 
highest amplitude emanate, the algorithm being such that 
30 when the two signals (Yl, Y2) of highest amplitude emanate 
from pixels in the same row or column a further signal 
processing step is undertaken to identify outputs (XI, X4) 
in said row and column from the two pixels respectively 
next adjacent to the pixels having the highest and lowest 
J5 signal amplitudes (Yl, Y4), and to identify the mean (M) of 
signal outputs from the eight pixels (Xc) diagonally 
adjacent to, and arranged around, said four pixels (Yn, Ye, 
Ys, Yw) in said row and column, to enable an amplitude 
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comparison to be made according to the result of which one 
of the said median pair (Y2, Y3) is selected. 

15. A signal processing method for estimating luminance for 
5 use in a matrix array monochrome image sensor in which 
elected luminance-sensing pixels (P) capable of producing 
signals representative of luminance are arranged in a 
chequerboard pattern with non- luminance-representative 
matrix array elements (N) which do not produce signals, the 
10 method comprising the steps of: ranking the signal values 
(Yl, Y2, Y3, Y4) representative of luminance obtained from 
a cluster of four luminance-sensing pixels (Yn, Ye, Ys, Yw) 
surrounding a first non-luminance-representative matrix 
element (C) , analysing the spatial distribution of said 
15 ranked signal values so as to detect the presence of any 
diagonal luminance contour edge, horizontal luminance 
contour stripe or vertical luminance contour stripe at said 
first non- luminance-representative matrix element (C) , and 
determining an estimated discrete signal value 
20 representative of luminance for said non-luminance- 
representative matrix element (C) based on one or more 
middle ranked signal values (Y2, Y3) in a contour-enhancing 
fashion dependent upon, firstly, whether any diagonal 
luminance contour edge is detected and, secondly, whether 
25 any horizontal or vertical luminance contour stripe which 
is detected is interpreted as a dark stripe on a light 
background or a light stripe on a dark background, wherein 
when a horizontal or vertical luminance contour stripe is 
detected the signal values representative of luminance 
30 obtained from at least second and third luminance-sensing 
pixels (XI, X4) disposed respectively within and outwith 
the detected luminance contour stripe are compared with the 
mean (M) of the signal values obtained from a plurality of 
further luminance-sensing pixels (Xc) in the vicinity of 
35 said cluster of four luminance-sensing pixels (Yn, Ye, Ys, 
Yw) so as to determine whether said horizontal or vertical 
stripe is interpreted as a dark stripe on a light 
background or a light stripe on a dark background. 
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16. A signal processing method according to claim 15, 
wherein said plurality of further luminance-sensing pixels 
(Xc) comprises eight pixels, each pixel being disposed 

5 diagonally adjacent one of said cluster of four luminance- 
sensing pixels (Yn, Ye, Ys, Yw) surrounding said first non- 
luminance representative matrix element (C) . 

17. A signal processing method according to claim 15 or 

10 claim 16, wherein said second and third luminance-sensing 
pixels (XI, X4) are disposed horizontally and vertically 
opposite said first non- luminance representative matrix 
element (C) and adjacent to the two of the cluster of four 
luminance-sensing pixels (Yn, Ye, Ys, Yw) which have the 

15 highest and lowest ranked signal values (Yi, Y4) . 

18. A signal processing method according to any of claims 
15 to 17, wherein the steps of the method are repeated for 
each non-luminance representative matrix element in the 

20 matrix array which is at the centre of both a row and a 
column containing seven matrix elements. 



19. A camera (20) comprising a matrix array image sensor 
(21) and signal processing means (25) formed and arranged 
for carrying out the signal processing method according to 
any preceding claim. 



30 



35 



20. A camera (20) according to claim 19 when dependent from 
claim 1 or claim 4, wherein said matrix array image sensor 
(21) comprises an array of red (R) , green ( G) and blue (B) 
pixels, and said elected luminance-sensing pixels are green 
pixels. 

21. A camera (20) according to claim 19 when dependent from 
claim 1 or claim 4, wherein said matrix array image sensor 
(21) comprises a mixed colour and monochrome array in which 
monochrome luminance-sensing pixels (Y) are arranged in a 
chequerboard pattern with colour-sensing pixels (R, G, B) . 
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22. A camera according to any of claims 19 to 21, wherein 
said matrix array image sensor (21) and said signal 

5 processing means (25) are provided together in an 
integrated circuit. 

23. A video camera system incorporating a camera according 
to any of claims 19 to 22. 

10 



15 



20 



30 



35 
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